HW3 Michal Grotkowski¶
I have trained two models: Random Forest and Logistic Regression, on the Steel Plates Fault dataset, which consists of 15 numerical features and a binary target. I choose two instances of each class and provide comparisons of LIME explanations of them below.
Instances of class 0¶
The comparison below depicts two different scenarios: a situation where models predicted different classes, and one where they depicted the same class. In the first scenario LIME explanation states that both models use values of the features V20 and V23 to shift the class prediction towards 0. There is a discrepancy, where Random Forest uses the value of V25 to shift the class prediction towards the true value of zero, while the Logistic Regression model does the opposite.
The LIME explanation of second instance for Random Forest is consistent for the value of feature V19 as it resides in the same range as the first instance.
This time both of the models predict the same class and the LIME explanation suggests that both of them use some of the feature values to predict this class, such as: V17, V23, V25, V26, V27.
print("Instance number %s. Class : %s \n" % (instance, targ))
print("Random Forest explanation:\n")
explainer.explain_instance(inst, predict_fn = forest_model.predict_proba).show_in_notebook()
print("Logistic Regression explanation:\n")
explainer.explain_instance(inst, predict_fn = log_reg.predict_proba).show_in_notebook()
print("Instance number %s. Class : %s \n" % (instance, targ))
print("Random Forest explanation:\n")
explainer.explain_instance(inst, predict_fn = forest_model.predict_proba).show_in_notebook()
print("Logistic Regression explanation:\n")
explainer.explain_instance(inst, predict_fn = log_reg.predict_proba).show_in_notebook()
Instances of class 1¶
This is the underrepresented class in the dataset and the figures below depict two different instances where the class predicted by two models differs.
In the first instance the Logistic Regression model predicts the correct class. This time Random Forest uses the value of the feature V23 to shift the class prediction towards 0, while Logistic Regression uses this value to shift it towards the correct value of 1.
In the second instance the situation is reversed: this time the Random Forest model predicts the correct class. LIME explanations here are interesting, as both of the models overlap in values of the features that are used to shift the prediction towards the correct class. Although for the Logistic Regression model we do not see the importance of the V16 feature, while for the Random Forest, its value shifts the prediction class towards the correct class.
print("Instance number %s. Class : %s \n" % (instance, targ))
print("Random Forest explanation:\n")
explainer.explain_instance(inst, predict_fn = forest_model.predict_proba).show_in_notebook()
print("Logistic Regression explanation:\n")
explainer.explain_instance(inst, predict_fn = log_reg.predict_proba).show_in_notebook()
print("Instance number %s. Class : %s \n" % (instance, targ))
print("Random Forest explanation:\n")
explainer.explain_instance(inst, predict_fn = forest_model.predict_proba).show_in_notebook()
print("Logistic Regression explanation:\n")
explainer.explain_instance(inst, predict_fn = log_reg.predict_proba).show_in_notebook()
Appendix¶
!git clone https://github.com/adrianstando/imbalanced-benchmarking-set.git
!pip install lime
import pandas as pd
import lime
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, balanced_accuracy_score
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
df = pd.read_csv("./imbalanced-benchmarking-set/datasets/steel-plates-fault.csv")
target = pd.get_dummies(df["TARGET"], drop_first = True)[2]
df = df.iloc[:,1:-1]
X = pd.get_dummies(df, drop_first = True)
X.head(), target.head()
forest_model = RandomForestClassifier(n_estimators = 100, max_depth = 10)
forest_model.fit(X, target)
log_reg = make_pipeline(StandardScaler(), LogisticRegression(max_iter = 1000))
log_reg.fit(X, target)
print(accuracy_score(target, forest_model.predict(X)), balanced_accuracy_score(target, forest_model.predict(X)))
print(accuracy_score(target, log_reg.predict(X)), balanced_accuracy_score(target, log_reg.predict(X)))
from lime.lime_tabular import LimeTabularExplainer
explainer = LimeTabularExplainer(X.values, feature_names = X.columns, training_labels = target)
import numpy as np
print(np.where(target == 1))
instance = 1401
inst, targ = X.iloc[instance], target.iloc[instance]
import warnings
def warn(*args, **kwargs):
pass
warnings.warn = warn
explainer.explain_instance(inst, predict_fn = log_reg.predict_proba).show_in_notebook()